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Abstract 

Introduction. The purpose of this qualitative, exploratory study is to clarify ambiguous 
concepts in intelligence services literature specifically related to measurement of 
intelligence outcomes and impact. 

Method. Face to face interviews were held with five subject experts from various 
intelligence fields and countries regarding their conceptualizations of intelligence 
measurement. Participants were given the opportunity post-interview to review and edit 
responses, to ensure accuracy. 

Analysis. Participant responses were compared and contrasted in defining key terms, 
descriptions of current practice in outcome measurement, and requirements for future 
best practice. 

Results. Participants' definitions of terms and conceptual lenses varied. Participants' 
descriptions of their own practices in, and reactions to, measuring outcomes and 
impacts were unique to the participant and paralleled discussions in the literature. 
Suggestions for best practice were made. Participants agreed that improvements to 
current measurement practice are both possible and necessary, and called for 
conceptual developments in intelligence measurement. 

Conclusions. The authors intend this study to provide a starting point for comparative 
discussion regarding intelligence measurement, and the need for conceptual 
developments, in a field that has historically been dominated by prescriptive 
measurement models and unique accounts of measurement practice. 
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Introduction 


Literature of various intelligence fields, such as competitive intelligence and business intelligence, 
contains criticisms regarding measurement practices and activities to determine the value of 
intelligence. These criticisms include concerns about the lack of conceptual consistency, the 
multiplicity of measurement approaches and lack of consensus regarding best practices, and the lack 
of research evidence to support measurement activities, both conceptually and methodologically (e.g., 
Hughes, 2005; Lonnqvist and Pirttimaki, 2006 ; Marin and Poulter, 2004 ; Wright and Calof, 2006) . 

As part of a larger research project, the authors examined the literature of intelligence measurement in 
a variety of intelligence fields, specifically measurement to determine organizational outcomes and 
impact of intelligence. The authors found, however, that the literature contained some conceptual 
ambiguities specifically related to outcomes, impact, and measurement. Note that for the purposes of 
this article, which deals with measurement of intangibles, the following definition is provided: 
Measurement is the activity of identifying and then quantifying phenomena on an accepted scale of 
quantity or value. 

Just as authors such as Savolainen (2009) and Bates ( 2005) have recognized the existence of 
ambiguous concepts in library and information studies and worked to clarify those concepts for the 
field, the authors determined that there was a need to go beyond what is explicit and more critically 
examine the underlying theory and philosophy of measurement being propounded in the intelligence 
measurement literature. Believing that consensus in terminology, conceptualization, and measurement 
practice is necessary in order to improve measurement and formulate best practices, the authors 
developed this exploratory and qualitative study. The study design consists of interviewing a small 
sample of intelligence experts working, publishing and presenting on intelligence measurement in 
various countries. Participants were asked: to define some key terms for outcome measurement; how 
they conceptualize intelligence measurement; if those conceptualizations address the challenges of 
outcomes and impacts in measurement, and if so, how; and the characteristics of an effective 
intelligence outcome measurement model. The interview guide can be found in Appendix A . 

The purposes of this study were: to obtain information about expert opinions and conceptualizations 
in order to clarify ambiguous concepts in intelligence specifically related to intelligence outcomes, 
impact, and measurement; and to initiate comparative discussion about intelligence measurement, 
which has been to date elusive in the literature. 

This article reviews intelligence measurement literature to identify conceptual challenges and the 
literature which informed the study design. A summary of the research design is then provided, along 
with the study findings. The discussion compares study participants' conceptual frameworks, 
providing an opportunity to make more explicit to researchers the challenges and potential future 
directions in conceptualizing measurement of intelligence outcomes to prove value or effectiveness, 
an important step toward consensus and the formulation of best practices. 

The conceptual problems of measurement for intelligence 

Competitive intelligence, which can be defined as the intense production and analysis of information 
about the competitive environment resulting in intelligence products for use in organizational decision 









-making, is one of several intelligence fields. Competitive intelligence, business intelligence, market 
intelligence, covert intelligence, counter-intelligence, and military intelligence share common roots 
and many practices (Juhari and Stephens 2006) . For all of them, the intelligence cycle involves the 
following steps: information needs are identified; data and information are sourced; data are analysed 
and become intelligence; and resulting intelligence products, such as reports and industry profiles, are 
distributed. Intelligence products are used to inform decision-making and thereby improve decision 
outcomes, which is the ultimate purpose of intelligence ( Bouthillier and Shearer. 2003 : Leslau. 2010 ). 
Intelligence is therefore a process, the intelligence cycle, and a product, the intelligence deliverables. 

Lonnqvist and Pirttimaki (2006) state that from their survey of business and competitive intelligence 
literature two primary reasons for measurement are given by authors: to improve process and to prove 
value. From a practical perspective, the value of measuring competitive intelligence is that it fosters 
improvement by providing data on performance (e.g., Blenkhom and Fleisher. 2007) . Authors have 
argued that measurement gives reassurance to high level stakeholders that there will be a return on 
investment, while staff, knowing that measurement activities are taking place, have an incentive to 
improve processes and deliverables (e.g., Buchda. 2007 ). Surveys have indicated that while there is 
little use of competitive intelligence performance measures in organizations (e.g., Marin and Poulter, 
2004 : Prescott and Bharwai, 1995 ). competitive intelligence practitioners are aware of the need of 
measurement to advance practices in their field (e.g., Qingjiu and Prescott, 2000) . 

There is a documented history of problematic measurement in all intelligence fields, and this may be 
due to a variety of factors. Competitive intelligence practices are a recent development, formalized 
with the publication of Michael Porter's book Competitive Strategy in 1980. The youth of the field, 
connected to not only limited measurement practices but also limited awareness of measurement 
needs, may partially account for its measurement challenges. For example, Herring's (1996) small 
survey of senior executives overseeing a competitive intelligence programme indicated that while 
none of the executives in his sample were evaluating their programme activities, they were open to 
doing so once the idea was introduced to them. 

In the field of national security intelligence (encompassing covert, military, and counterintelligence), 
Dahl (2010) argues that performance measurement has historically been a simple success or failure 
binary that does not provide critical information for improving performance. Reasons given in the 
national security intelligence literature for this narrow view of measurement tend to revolve around 
methodological challenges related to the politicization of intelligence in national security settings 
(Hulnick, 2006) and problems with conceptualization of measurement (Hastedt, 1991; Turner, 1991) . 

Another complicating factor for all intelligence fields may be that measurement itself is inadequately 
understood and practiced. Authors in a variety of fields related to business and the social sciences 
have argued that measurement practice does not follow measurement theory strictures and tenets, 
resulting in data and instrumentation errors (Bontis, 2001 ; Flamholtz, 1980 ; Gorad, 2010; Pike and 
Roos. 2004) . 

Issues of practice, history, and politicization aside, significant conceptual challenges which are 
retarding development of effective and standardized measurement approaches and tools are identified 
within the literature. These conceptual challenges to outcome measurement can be summarized as 
follows: 


Multiplicity of measurement models, approaches 






















There are many measurement models and recommendations in the literature (e.g., Davison, 2000 ; 
Herring. 1996: Moore. Krizan and Moore. 2005 : Steventon, Jackson. Hepworth. Curtis and Everitt. 
2012) but comparative discussion regarding the models and recommendations is elusive. 
Recommendations tend to be prescriptive rather than tested. As Blenkhom and Fleischer (2007) have 
pointed out, much of the literature consists of anecdotal reports of practitioner experience, which does 
not provide the evidence necessary to formulate best practice. Of course, many authors have also 
argued that the unique situation of each industry and organization renders the standardization of 
measurement tools impossible (e.g., Kilmetz and Bridge, 1999 ; Rothberg and Erickson, 2005) . 

Inconsistent use of terminology 

Notable among challenges for intelligence measurement is the significant conceptual inconsistency 
across authors and publications in describing measurement and value, as found, for example, by 
Wright and Calof (2006) in their small study examining published competitive intelligence research in 
three countries. Inconsistencies between organizations and agencies in terminology use also exist; the 
discrepancies in use of the terms business intelligence and competitive intelligence in various parts of 
the world are an example (Buchda, 2007) . 

Unproven outcomes 

Lonnqvist and Pirttimaki (2006 , p. 33) complain of output value conceptualizations, stating that the 
literature 'includes a lot of unverified assumptions' about the possible benefits of intelligence to 
organizations. A comparison of the many surveys which ask competitive intelligence practitioners and 
managers what benefits should result from competitive intelligence contains a great deal of overlap, 
but also significant discrepancies (see Hannula and Pirttimaki, 2003; Jaworski and Wee, 1992; Marin 
and Poulter, 2004 : Oingiiu and Prescott. 2000 ). Such speculative responses are at least partially due to 
a lack of research providing causal evidence as to what outcomes, beneficial or otherwise, are related 
to intelligence practices. 

Good enough measurement 

In his textbook on security and military intelligence analysis, Clark (2010) calls for 'the right things' 
to be measured, pointing out that intelligence organizations in the US and Russia do not have 
established feedback loops to provide assessment of how valuable intelligence activities have been. 
Instead, the measurement focus tends to be on process. Yet the purpose of intelligence is not process. 
For all intelligence fields, the primary purpose of intelligence is to inform decision-making, with the 
intent to increase the likelihood of the most optimal outcomes for the organization (Bose, 2008 ; 
Leslau, 2007; Turner, 1991) . This disconnect, rooted in the significant conceptual and methodological 
challenges of not only measuring intelligence outcomes but intangibles measurement itself, has led to 
the popularity of good enough measures that do not give a complete picture of value to the 
organization or performance of the intelligence unit. Measures in use may be the accuracy and 
timeliness of the intelligence deliverables, and numbers of reports delivered over time. 

Such process measures, which treat intelligence as an output of a system rather than an input into 
decision-making, in combination with user satisfaction surveys, appear to be the foundation of good 
enough metrics in both national security and business-related intelligence fields (Ganesh, Miree and 
Prescott, 2004; Moore et al. 2005) . Unfortunately such process measures do not address problematic 
questions of validity in performance or value measurement. Determinations of value or benefit are 
necessary components in conceptualizations of performance. 

























Performance can be defined as a set of criteria critically applied to purposeful activity within 
organizations. Carton and Hofer 12006 : 3) in their handbook on measuring organizational 
performance called it a "contextual concept" and a "multi-dimensional construct" that "involves 
measurement of the effects of organizational actions". Performance varies by industry and sometimes 
organization, determining what results in relation to which situational values are good or bad. 
Situational value judgments aside, the definition provided points to the essential criterion for 
determining performance, an identification and quantification of effect, or outcome and impact, which 
process measures alone cannot provide. 

Calls for improvement 

Scholars who have recognized the conceptual and methodological problems inherent in current 
measurement practices have made calls for research. Those interested in determining the value of 
competitive intelligence have made calls for research so that measures of competitive intelligence 
outcomes might be developed and that the benefit and value of it to organizations might be 
determined (Hughes, 2005 ; Marin and Poulter, 2004 ; Wright and Calof, 2006) . In the national security 
field, calls have also been made for better conceptualization and, in turn, communication of what 
intelligence is and what it can be expected to accomplish (Lowenthal. 2013: Turner. 1991 V 

In the light of the problems and discussions identified above, and recognizing that, as argued by 
Liebowitz and Suen (2000 1. conceptual disagreements and problems lead to measurement confusion 
and error, the authors developed a study to try and address some of the conceptual questions about 
intelligence measurement. 

In reading the literature of intelligence measurement, it became apparent to the authors that some of 
the most interesting discussion about measurement was being made implicitly, in statements that only 
half-revealed discussions about measurement and intelligence and hinted at the authors' own 
conceptual lenses. These half-visible discussions raised significant questions. For example: why do 
some authors, many of whom are practitioners, make the statement that standardization of 
measurement tools is impossible for intelligence work? Tools themselves, whether they be Likert 
scales or sliding rulers, once accepted and understood as a scale and indicator of size or value, can be 
used regardless of context. Do these authors actually mean that a suite of tools with customizable 
interchanging parts, dependent on organizational context, was necessary? Do these statements reflect 
a belief that measurement would be essentially process-related and not outcome-based? Were these 
statements presupposing unique tangible outcomes and ignoring more generic intangibles under 
headings such as innovation, relationships, or policy? 

Authors in the intelligence measurement literature may provide only partial explanations as to the 
rationale behind their recommended measurement models and approaches, and there is little 
discussion as to how one author's recommendations align with, complement, or refute those of another 
author. Such elliptical discussion in the literature is likely a contributing factor to the conceptual 
difficulties of intelligence measurement. There is also little discussion of measurement theory and 
best practices in measurement when making recommendations, and of what good measurement looks 
like. 

Recognizing the common roots and practices (namely, the intelligence cycle) of all intelligence fields, 
the authors decided to interview subject experts, people who were presenting and publishing in 
intelligence measurement in various intelligence fields, in order to examine in closer detail 
conceptualizations of intelligence measurement that are informing research and practice and explore 









whether the elliptical references in the literature (referenced above) would merit further study. 
Sampling criteria were that at least three different intelligence fields were represented by participants; 
academics and practitioners would be included; and at least two countries would be represented in the 
sample. This sampling criteria was developed in order to mirror the broad range of intelligence 
measurement literature which prompted the research questions, and to provide opportunity for a range 
of divergent opinions, in the belief that where or if those opinions converged, it might provide insight 
into potential areas of consensus. 

Because this study was developed to meet a need within a larger, doctoral study concerned with 
measuring organizational outcomes and impacts of intelligence, the research questions were 
formulated as follows: 

1. How do these experts conceptualize the outcomes and impact of intelligence? 

2. How do experts in intelligence conceptualize measurement for intelligence 
outcomes and organizational impacts? 

3. How do the measurement models used and/or developed by these experts compare 
to one another? 

The purpose of this small exploratory study was to clarify discourse in the literature around 
intelligence measurement, specifically outcome-based intelligence measurement. 

Research methods 

The authors conducted a field study in 2012, interviewing a sample of five intelligence experts 
working in Finland, the United Kingdom, and the United States. Experts were defined for the 
purposes of this study as scholars and practitioners in any field of intelligence (competitive, market, 
military, covert, etc.) who have publicly presented or published on intelligence measurement. 
Participants came from the fields of business intelligence, competitive intelligence, and covert/counter 
intelligence. They included university faculty, former US Central Intelligence Agency employees, and 
a competitive intelligence practitioner working with the UK Competitive Intelligence Forum to 
develop performance measures for competitive intelligence. 

Potential participants were identified through publications and conferences, and snowball sampling 
was used in an attempt to increase the number of participants. Interviews were face to face, semi- 
structured interviews, which took on average just under an hour. Prior to the interviews, participants 
were sent a copy of the interview questions along with informed consent forms. All participants were 
notified that they could refuse to be recorded, participate anonymously, refuse to answer a specific 
question, or withdraw at any time, including after the interview was concluded. All participants 
completed the interview and agreed to have their names published in this study, as authors and 
presenters of research and/or practice in the field of intelligence and measurement. The five 
participants were: 

Professor Antti Lonnqvist, Department of Information Management and Logistics, 

Tampere University of Technology, Finland 

Dr. Sheila Wright, Leicester Business School, De Montfort University, UK 
Mr. Andrew Beurschgens, Head, Market and Competitive Intelligence at a large UK 
mobile telecommunications firm; Board Member for the UK Competitive Intelligence 
Forum (UKCIF) 

Dr. Stephen Marrin, Centre for Intelligence and Security Studies, Department of Politics 



and History, Brunei University, UK 

Dr. John Kringen, Researcher at the Institute of Defense Analyses; formerly of the US 
European Command and the US Central Intelligence Agency 

Participants were asked to comment on their measurement conceptualizations and models in relation 
to intelligence and outcomes rather than processes. They provided their own definitions of the terms 
output, outcome, and impact', descriptions of their measurement tools and measurement methods; and 
critiques of current intelligence measurement research and practice. They also provided some 
description of the history of their conceptualizations and their measurement models, and some 
informed critique of each model's purposes, strengths, and weaknesses. 

After the interviews, the researcher reviewed the interview notes and recordings to capture notable 
quotations and to partially transcribe responses of participants. Reponses were summarized and 
compressed to distil essential elements for comparison. These summarized notes were then sent in 
electronic, written form to study participants, who were given the opportunity to review and edit the 
researcher's notes and conclusions to ensure that the notes accurately reflected their thoughts, 
practices, and comments on intelligence measurement. This review served the additional purpose of 
preventing publication of any inaccurate statements participants might then wish to retract. 

Once all participant edits were received, the data were analysed. Responses were compared and 
contrasted to one another in several categories, as shown in the next section, which reviews the 
findings. 

Findings 

The findings of the study are here grouped under three subheadings: definitions, descriptions of 
participants' current outcome measurement practices, and description of ideal outcome measurement. 

Definitions 

Participants were asked to provide definitions for three terms: output, outcome, and impact. Some 
definitions were closely aligned, while others varied widely. 

All participants agreed that outputs are usually tangible and easily identified. Outputs may take the 
form of products such as reports, increased situational awareness or knowledge on the part of the 
intelligence recipient, subsequent actions, or events. Outputs were described as typically occurring 
soon after the intelligence has been delivered to an audience. 

While all participants agreed that outcomes were distinct from outputs and typically intangible in 
nature, some significant discrepancies occurred in the definitions provided. Outcomes were described 
by four of the five participants as an effect or intangible change within the decision-maker, the 
decision itself, or the audience who received the intelligence. Such outcomes might be a decision¬ 
maker's perspective, the effect on the decision itself, or the changed information need or information 
reception due to intelligence provided after a decision is made. 

Antti Lonnqvist and John Kringen described outcomes at an organizational level and as building on, 
and being related to, outputs. Sheila Wright and Stephen Marrin stated that outcomes and outputs are 
unrelated and that the provision of intelligence may result in outputs, outcomes, both outputs and 
outcomes, or neither. Andrew Beurschgens defined outcomes as the intangible effects upon the 



audience, asking, Is the audience stimulated, provoked, motivated? For him, this is the outcome of 
intelligence, and it is directly related to the salesmanship of the competitive intelligence practitioner; 
in other words, his/her ability to get an audience engaged in using intelligence. 

The greatest discrepancies in definitions occurred around the concept of impact. All agreed that 
intelligence impact is not, and should not be defined as, related to organizational strategy, although 
Antti Lonnqvist and John Kringen stated that the decision-maker(s) may link intelligence to strategy. 
Andrew Beurschgen did offer a caveat that if a competitive intelligence programme is aligned with 
corporate strategy then it is expected to influence the outcomes of strategic reviews. 

Impact was variously defined as reduction of risk in the decision-making process (Sheila Wright), its 
effect upon policy (Stephen Marrin and John Kringen), its effect upon the decision-maker in the 
context of a decision (John Kringen), an indication of success that is closely related to, perhaps 
synonymous with, outcomes (Antti Lonnqvist), and simply, the magnitude of a given outcome's 
influence (Andrew Beurschgens). 

Current measurement practice 

Participants were asked to describe how they currently measure intelligence outcomes and impact. 
Only three of the five participants reported that they have attempted to measure outcomes and impact. 

Antti Lonnqvist described his outcome measurement approach as a generic model often found in 
business literature and used by other scholars. This generic model consists of direct, indirect, 
subjective, and objective measurement, and the need to take pre-measurement steps. The pre¬ 
measurement steps are questions which ask why there is a need for measurement, what is being 
measured, and identify success factors and standards in relation to the input(s) and the viewpoint(s) of 
the audience. While he sees conceptual and practical limitations in this model related to identifying 
success factors that address contextual variation, he sees its strength as its customizability and thus 
how customization forces users to consider the purpose of the measurement activity, thus increasing 
its validity. 

Antti Lonnqvist, in common with other participants, indicated that in his measurement approach, 
process measures are necessary to inform and make possible outcome measures: 

Anyone trying to measure the outcomes of intelligence needs to understand the 
intelligence process and what is on the manager's mind when requesting, using, and 
discussing intelligence. Then we can identify what new information is brought by the 
intelligence. For example, before we can measure impact we have to ask, is the 
information being accessed and then being used? So a process measure such as usage 
statistics is needed to be a part of outcome measurement. 

Andrew Beurschgens and John Kringen described customer feedback as the chief mechanism through 
which outcomes and impact might be assessed, if not measured. John Kringen stated that general 
practice at the CIA is to use a combination of process and satisfaction measures in combination with 
debriefing sessions to obtain insight into how well their service is valued by their users. Although he 
acknowledges it to be imperfect, he also considers this practice to be useful and workable. As a part of 
these measures the CIA attempts to identify indicators {signposts) of success. However, such 
feedback is not quantified by any kind of formal metric, and is not always available, due to 
problematic access to users, particularly high-level decision-makers. Similarly, Andrew Beurschgens, 
describing practices in competitive intelligence, states that he uses a high level structured feedback 



approach, based on the work of Tim Powell (a competitive intelligence practitioner and author of how 
-to texts such as Analyzing Your Competition: Its Management, Products, Industry and Markets, 
published by Find/Svp Info Clearing House) and research being done with the UK Competitive 
Intelligence Forum. Such user feedback, in his view, should relate to questions about the timeliness 
and usefulness of the intelligence, whether the stakeholders were better-informed about relevant 
issues, and whether the decision-makers were better enabled to reach a consensus. He also described 
how technology tools, through tracking processes such as usage rates and new project development, 
might also identify outcomes and impacts such as the dollar value of business opportunities lost or 
gained. 

Stephen Marrin and Shelia Wright do not have an outcome measurement tool or method that they use, 
although Stephen Marrin (2012) has conceptualized directions intelligence measurement could take, 
such as batting averages. Sheila Wright stated in her interview that if she were asked to measure the 
outcomes or impact of competitive intelligence, she would attempt to convince the requestor not to try 
the nearly impossible. She then continued with this statement, questioning the value of such 
measurement: 

Rather than ask, what is the value of having Cl, it is more useful to ask, what is the value 

of not having Cl? Another question is, why do we need to prove the value of Cl units? 

There are many business departments, like strategic planning, which are considered just 

"a cost of business", which are not required to prove their value. 

All participants agreed that intelligence measurement as it is currently practiced, including their own 
measurement practice, is problematic and could be improved. Responses given as to why these 
problems exist were voluminous and diverse. Participants were unified in citing problems with 
managing feedback mechanisms, namely gaining access to intelligence users, and the subjectivity of 
user statements. Participants also cited problems with establishing causal relationships between action 
and effect in intangibles, and isolating effects for measurement. 

Significant conceptual problems for current measurement practices were also discussed. One was the 
lack of consistency in approach to measurement, which is directly related to non-standardized 
measurement tools. Another is attitudinal: under-valuation of both intelligence and measurement 
resulting in non-cooperation between departments, and managerial resistance or disinclination to 
participate, in organizations. 

Stephen Marrin argued that a conceptual framework is needed for security intelligence measurement, 
stating fundamental concepts that would inform intelligence measurement are not yet developed. As 
an example, at one point in his interview, he cited the fact that while in business financial measures 
can be used as a fundamental quantifier of value, there is no single currency of value in intelligence. 

John Kringen pointed out, specifically in relation to security intelligence, that the weakness of current 
intelligence measurement is that there is no conceptual model of intelligence system dynamics that 
looks at both inputs and outputs, when outputs are policy outcomes either domestically or in recipient 
societies. Such conceptual models, he argues, are necessary in order to determine feasible outcome 
measurement. He discussed the need for ways to quantify intelligence challenges in the context of the 
intelligence problem. He noted that rating intelligence by its accuracy is not an adequate reflection of 
performance. Very simple research tasks might result in perfect accuracy, while highly complex and 
challenging research tasks involving multiple stakeholders might result in less accurate, but 
potentially far more valuable, intelligence products. He suggested that rating intelligence performance 
should be more like assigning scores to Olympic diving than generating batting averages in baseball, 



but acknowledges the conceptual challenge to such a shift in perspective and measurement tools is 
that right now there is no agreement on the judging scales to be used. 

These conceptual criticisms were echoed by Andrew Beurschgens, who stated that not only his own 
measurement approach, but all current competitive intelligence measurement approaches, lack 
consistency and rigour. He attributes these problems to inadequate research into and literature on 
conceptual models upon which measurement might be based, commenting that 'there is not the same 
level of literature available on measurement models as there is now on the analysis part of the Cl 
process'. 

Future measurement 

All participants acknowledged that their measurement practices and conceptualizations were 
imperfect, and stated that they considered their measurement approach to be dynamically changing as 
they encountered new research and ideas for practice. Participants were asked to comment on what 
outcome measurement should be in the future, specifically naming desirable characteristics of an ideal 
robust and useful outcome measure for intelligence. Four of the five participants supplied 
homogenous lists of necessary characteristics. 

According to those responses, any robust and useful measure of intelligence outcomes would be: 

• Reliable: not only meaning that the measurement tool be consistent, but that more than one 
measurement tool is used in a composite or multi-measurement method approach 

• Valid: the audience and purpose of the measurement activity are addressed and made explicit 
by the choice of measurement tool(s) and approach(es) 

• Causal: the measure relates intelligence to beneficial effects (traces causal relationships) 

• Credible: results obtained are supported by evidence of value, either quantitative or qualitative, 
positive or negative. The data captured is non-politicized and objectively fair 

• Usable: the measure is not only easy and simple to use but also to understand, fostering 
communication between measurer and audience 

Other noteworthy but disagreed-upon characteristics and elements were suggested. Sheila Wright, 
who defines competitive intelligence as behavioural change, stated that an outcome measure should 
provide evidence of behavioural change. Andrew Beurschgens, who has worked with Sheila Wright 
in UK Competitive Intelligence Forum, included self-help as an element for his proposed outcome 
measures, defining it as those who will seek out intelligence in response to an information need. His 
concept of self-help as an outcome could potentially be an outcome measure for competitive 
intelligence, when identified as an acquired behavioural response. He also believes, however, that 
measures can and should provide evidence of value to the organization as a whole. Along with Antti 
Lonnqvist he suggested that a suite of measurement tools should also include a financial measure. 

Antti Lonnqvists business management perspective and Sheila Wright's behavioural change 
perspective meant that their answers to the interview questions provided both intriguing contrasts and 
points of correlation. One such correlation occurred around the issue of precision and accuracy in 
measurement. Sheila Wright argued that it is impossible to measure intelligence value entirely and 
exactly. Instead, a measurement approach would have to accept that only elements can be captured, 
and these elements would provide a partial but adequate picture of value. In a discussion about the 
cost-effectiveness of measurement, Antti Lonnqvist pointed out that academic research looks for 
accuracy and can make extreme investments in highly complex measurement tools to ensure accuracy 
and to advance research. Business, in contrast, is often willing to compromise on the accuracy of 



measurement in order to keep costs down and to simplify measurement activities. He also stated that 
while accuracy is not possible with inaccurate phenomena such as intelligence outcomes, inaccurate 
measures can still be helpful, and so he does not see the inaccuracy of outcomes measurement as an 
obstacle. 

In other comments, Stephen Marrin discussed the need for outcome measures which might relate to 
the role intelligence plays in decision-making and need for proxy measures that could indirectly 
capture intangibles. Andrew Beurschgens suggested that a measurement model would need to allow 
for anecdotal evidence and account for the quick depreciation of deliverables, since a competitive 
intelligence product is often a single-use item. Several participants also discussed the need for 
outcome measures to show organizational (inter-departmental) usage and reflect varying stakeholder 
perspectives. 

These findings are discussed in the next section. 

Discussion 

The literature review identified certain criticisms and weaknesses of intelligence measurement that are 
supported by the findings of this study. Discussion of the findings is put into subsets, below, to 
facilitate comparison to the literature review. 

A multiplicity of models 

Some participants complained of the lack of consistency and research in intelligence measurement 
literature. Yet each described his or her own unique approach to or model of outcome measurement. 
This reflects what the authors have found in the literature, which is that while some approaches had 
elements in common, none were identical. Interestingly, while current practice was disparate, 
participants exhibited strong similarities in their conceptualizations of ideal measurement, discussed 
below. 

Inconsistent use of terminology 

When asked to define a sample of terms related to measurement, participants gave varied answers. 
Even when there was consistency amongst participants, a broader reading of the literature could give 
examples of other researchers who disagree with the definitions provided in this study. 

For example, participants agreed that impact should not relate to organizational strategy and that 
intelligence is not related to strategy. Yet within the field of competitive intelligence and business 
intelligence, researchers have identified organizational strategy (Herring, 1996) , strategy formulation 
(Hughes, 2005) , and strategic decision-making (Bose, 2008) as being strongly related to competitive 
intelligence and discovering its value in the application of competitive intelligence deliverables. 

The variations in terminology use found amongst study participants reflect larger inconsistencies 
within intelligence fields. 


Unproven outcomes 





Participants did not state a need for research to give evidence of intelligence outcomes. However, 
discussions of ideal outcome measures in the future described measurement tools that are able to 
make connections between the role of intelligence and improved outcomes, providing evidence of 
causality. 

Good enough measurement 

The literature review described a good enough existing measurement practice in intelligence; namely 
process measures combined with satisfaction measures. John Kringen stated that this measurement 
practice, although imperfect, is workable and sufficient to meet the most pressing needs in 
intelligence work at this time. However, all participants indicated that current measurement practices 
could be improved, and that they would welcome such improvements. 

Sheila Wright and Anitti Lonnqvist indicated that partial and inaccurate measurement can suffice, if it 
provides a simple and inexpensive method to obtain adequate understanding of the phenomenon 
measured. This concept of acceptable inaccuracy may be worth exploring. Gorad 1 20101 states in his 
article on measurement in the social sciences that an inaccurate measurement tool is acceptable for 
use if the margin of error is known to the user and the audience. Determining an acceptable margin of 
error would be useful in formulating intangibles measurement tools, and has not yet been explored in 
the intelligence literature. 

Calls for improvement 

The literature review cited studies calling for improvements to existing measurement practice. 
Interestingly, although the study participants acknowledged methodological challenges to 
measurement, such as access to decision-makers and the lapse of time necessary for outcomes to 
manifest, their strongest criticisms of current practice were reserved for conceptual problems. For 
example, the issue of no single currency in intelligence noted by Stephen Marrin was one of many 
made by participants which highlighted their concerns that inadequate or nonexistent conceptual 
models are significant challenges for developing effective intelligence measurement. 

Such discussion has been previously initiated in intelligence measurement literature, but to date it has 
been largely overshadowed by discussion of methodological challenges of measurement, in tandem 
with prescriptive conceptual models and frameworks. The emphasis placed by the participants on 
conceptual challenges as being the significant source of problematic measurement practice suggests 
that the time has come for the field to engage in the comparative conceptual discussions and 
formulation of consensus suggested by this paper. 

Pointers toward best practices 

Discussions in the literature of best practices in intelligence measurement are elusive. Yet when 
asked, the study participants were able for the most part to very clearly describe characteristics of 
ideal outcome measurement, even if they could not describe what the measurement tools themselves 
would look like. The checklist provided in the study findings will hopefully provide some fodder for a 
larger discussion about what intelligence measurement best practice should be, as the discrepancies in 
descriptions of best practices provoke a slew of additional questions not yet raised in the literature. 

For example, does valid outcome measurement require process measure components, as argued by 
Antti Lonnqvist? The checklist may also provide a useful starting point for subsequent research on 
best measurement practices. 



A variety of conceptual lenses 


In trying to understand the roots of so many different measurement models and approaches in the 
literature through discussions with study participants, it became clear that even those within the same 
field of intelligence did not agree on what intelligence is about, or what might be termed its about- 

ness. 

Below is a figure that attempts to visually represent the interpretive lens through which each 
participant appeared to speak. Participants might well disagree with this, but it is offered here not to 
pigeonhole them, but rather to show some of the varying stances possible and represented by the 
participants in the answers they provided specifically for this study. 


X Perspectives on the about-ness of intelligence 


Figure 1: Perspectives on the about-ness of intelligence 

As stated in the introduction, intelligence is sometimes defined as a piece of information (the 
intelligence product) destined to inform a decision-making process. Intelligence is also defined as a 
function or process within organizations. Participants appeared to represent differing viewpoints as to 
the role of intelligence as an organizational service or function to be valuated, and this can be seen in 
Figure 1. 

Sheila Wright stated that intelligence is about behavioural change: upon receipt of the intelligence, 
how is the audience affected? For example, how was the decision-maker influenced? Did his/her 
attitude toward a situation change? Stephen Marrin spoke of how intelligence is analysis: the analyst, 
how well the analyst performs, and the quality of the intelligence produced. Antti Lonnqvist took a 
management-level perspective, where the role of intelligence is a business function, intended to 
advance the goals of the larger organization. The other two participants occupy ground between two 
perspectives, with John Kringen leaning toward intelligence analysis, and Andrew Beurschgens 
leaning toward behavioural change (Figure 1). 

Such differences in perspective may be a significant factor in determining why there is a lack of 
standardized measurement practices in intelligence, and partially explain why there continues to be 
prescriptive measurement methods and models presented in the literature. Further research is needed 
to determine if more conceptual stances regarding the role of the intelligence function in organizations 
exist than those represented here. Such varied stances, unreconciled, will mean that measures will 



need to account for audience perspectives on the role of intelligence within the organization, and for 
its related considerations as to what outcomes are valued. 

Marrin ( 2012) has stated that there is no consensus about the purpose of intelligence. The authors of 
this study would expand on this and state that while the commonly accepted intelligence cycle 
suggests (and many studies have supported this) that the purpose of intelligence is to inform a 
decision, there is no consensus as to how the decision outcomes should be improved; i.e., to what end 
a decision is informed, which materially affects the would-be measurer's conceptualization of 
outcomes, impact, and their indicators. 

The tensions between desired measurement approach and conceptualizations of intelligence about- 
ness, are significant and need to be addressed. Antti Lonnqvist discussed in his interview how 
outcome measurement, once developed, will need to provide for different audiences. He gave an 
example of business managers who want financial quantities assigned to represent value and who 
consider subjective measures highly suspect. This presents a problem because qualitative measures 
are often subjective but are needed to quantify intangibles and cognitive effects. Turner (1991, p. 276) 
in describing the challenges for national security intelligence measurement, stated that 'responsiveness 
to the consumer is of utmost importance in evaluating intelligence as a governmental function', 
adding that '[Evaluating...intelligence is a complex task that must take into account organizational 
goals, bureaucratic imperatives, availability of resources, and the uses to which policymakers put the 
intelligence product' ( 1991 , p. 283). Such complexities are insurmountable if there cannot be basic 
conceptual agreement as to what outcomes of intelligence are expected, as a result of informing 
decision-making activities. 

Conclusion 

The study found that that while participants' definitions of outputs and outcomes aligned, definitions 
of impact were unique to each participant. Participants' descriptions of their own practices in, and 
reactions to, measuring outcomes and impacts were varied and paralleled discussions in the literature. 
These included philosophical questions about the need for measurement, necessary questions to 
determine the reliability and validity of the measurement, descriptions of methodological challenges 
related to identifying and quantifying intangibles, use of qualitative and quantitative tools, and 
process and satisfaction measures as a surrogate for outcome and impact measures. 

Overall, participant responses paralleled comments and discussions that can be found in the 
intelligence measurement literature, and provided some additional insights into those discussions. 
Conceptualizations of intelligence outcomes and impact varied with the perspective of the participants 
as to the function of intelligence in the decision-making process. 

Participants agreed that significant conceptual and methodological challenges exist for developing 
intelligence outcome and impact measures, and argued for research into addressing the conceptual 
challenges as a priority for the development of effective measurement. Although they were unsure 
what future measurement tools might be, and some questioned whether such measurement is possible, 
most gave detailed and sophisticated lists of criteria which would need to be met by a successful 
measure. Such a list is unique in the literature and in conjunction with the conceptual differences on 
the part of the respondents would merit further study. 

The value of this study is shown in several ways. First, the insight it provides into the conceptual 
frameworks used by experts in developing, testing, and describing current intelligence measurement 





practices. Second, the comparison of these frameworks provides an opportunity to make more explicit 
to researchers the challenges and potential future directions of intangibles measurement, specifically 
as they relate to intelligence outcomes in proving value or effectiveness. Third, the study is a step 
towards opening discourse to determine what commonly used terms mean to various intelligence 
groups, so that conceptual challenges to measurement can be addressed. The authors hope the 
juxtaposition of participants' current measurement practices with their critiques of the field and 
criteria for future measurement best practices will provide a valuable starting point to begin this 
conceptual discussion, which study participants indicate is needed in order to move research and 
practice forward. 

This study will be followed up with additional research to determine the feasibility of outcome-based 
measurement for competitive intelligence services within organizations, where competitive 
intelligence is used in organizational decision-making processes. Additional research may include a 
survey of intelligence practitioners in various fields to determine how closely the comments of the 
experts in this study reflect the conceptualizations and practices of the larger field. 

Sheila Wright and Stephen Marrin stated in their interviews that the current state of intelligence 
measurement is so problematic that it is essentially untaught in their university courses. This holds 
implications for the practices of graduates. We cannot be surprised that other audiences, whether they 
be managers or taxpayers, undervalue what is done in any intelligence field, when we as researchers 
and educators are unable to cogently express its value through comprehensible measurement. It is the 
hope of the authors that in providing a starting point for conceptual comparisons, this study may 
prompt conversations between researchers, students, and practitioners in this field that may lead 
toward consensus of terminology and conceptualization, research to investigate what outcomes should 
be used to supply data for measurement of intelligence activities, and formulation of best practice 
through systematic testing of measurement models and recommendations in the literature. 
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Appendices 

Appendix A: Interview Guide 

1. How do you conceptualize competitive intelligence (business intelligence, etc.) outcomes? 

2. How do you differentiate outcomes from outputs? 

3. How do you conceptualize competitive intelligence impact(s)? 

4. Do you see impact(s) as being tied to organizational strategy? 

5. How do you (or would you) measure outcomes and impacts? What does this measurement 
approach/method/model look like? 

6. Where does your approach/method/model of intelligence measurement come from? 
(development of model - history, perceived need, anticipated use) 

7. How would you describe the methodological challenges related to Cl and: 














a. Informed decision making? 

b. Intangibles? 

c. Qualitative data? 

d. Time lag? 

e. Any methodological challenges you have found? 

8. How would you characterize a robust and useful measure for competitive intelligence? What 
attributes would the perfect model possess? 

9. Do you perceive your approach/method/model as having strengths and/or weaknesses not 
present in other approaches/methods/models of measurement? 

10. Do you see your conceptualization as static or dynamic? Is there room for changes and edits 
going forward? If yes, what are they likely to be? 

11. How do you share this approach/method/model with others? 

12. Do you know of any other experts in this area which might be willing to participate in this 
study? 
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